Behavior of message_length_limit

• When the byte-size of the buffer exceeds the message_length_limit, the buffer will be cleared.
• If there is subsequent data, that part is discarded so that a warning of parsing failure doesn't occur.

Run Fluentd with the following setting.

<source>
  @type tcp
  tag test
  message_length_limit 12
  delimiter ";"
  <parse>
    @type json
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send data as follows.

$ netcat 0.0.0.0 5170
{"message":
"hello
world!"};
{"k":"v"};

Result:

[info]: #0 The buffer size exceeds 'message_length_limit', cleared: limit=12 size=19 head="{\"message\":\n\"hello\n"
(datetime) test: {"k":"v"}

Normal behavior

Run Fluentd with the following setting.

<source>
  @type tcp
  tag test
  delimiter ";"
  <parse>
    @type json
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send data as follows.

$ netcat 0.0.0.0 5170
{"message":
"hello
world!"};
{"k":"v"};

Result:

(datetime) test: {"message":"hello\nworld!"}
(datetime) test: {"k":"v"}

Writing to a socket at high speed can cause multiple writes to be stored in the buffer before the server reads them.
The new tests are exactly those.

It is undesirable that high-speed writing causes the total buffer size to exceed the limit and drop the records even if each record size does not exceed the limit.

So I fixed to check the size at 2 points: delimited one message size and total remaining buffer size.

One concern is whether this fix will affect the performance improvement of the next fix.

• in_tcp: Improve the performance for multiple events case #2567

Thanks for your review!